ED 341 684 



SP 033 575 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



P^JB TYPE 



Hogge, James; Murrell, John 

Exploring Values Underlying the Assessment of 

Teaching Competence: An Application of Social 

Judgment Theory* 

Nov 91 

24p.; Paper presented at the Annual Meeting of the 
Mid-South Educational Research Association 
(Lexington, KY, NovemCer 15, 1991). 
Speeches/Conference Papers (150) — Reports - 
Research/Technical (143 ) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 

Competence; Cooperating Teachers; Elementary 
Secondary Education; •Evaluation Criteria; 
•Evaluation Problems; ^Evaluation Research; Higher 
Education; ^Individual Differences; ^Judgment 
analysis Technique; Student Teacher Evaluation; 
Student Teacher Supervisors; « value Judgment 
•Fairness 



ABSTRACT 

This study was conducted to examine individual 
differences with respect to values underlying the assessment of 
teaching competence in order to enhance the ef f ectivenesr and 
fairness of assessment. Nine classroom teachers who were preparing to 
be mentors of beginning teachers and five teacher educators indicated 
the relative Importance (expressed values) chey attached to the 
following aspects of teaching: teacher/pupil relationships; 
preparation and management; curriculum content; classroom 
interaction; assessment and records; and self-evaluation. They then 
rated the competence of 50 hypothetical student teachers for whom 
scopes were available. Relative weights (implemented values) applied 
to the aspects of teaching were computed for each participant by 
employing a regression model, utilizing scores as predictors and 
participants' overall ratings as criteria. Results found individual 
differences with respect to both expressed and implemented values # 
and several participants exhibited discrepancies t^tween expressed 
values and implemented values. The latter result may reflect a lack 
of self-awareness and could adversely affect communication among 
assessors and between assessor and assessee. (Author/LL) 



* Reproductions supplied by EDRS are the best that can be made 
» from the original document. 



Exploring Values Underlying the Assessment of Teaching Coapetence 
An Application of Social Judgment Theory 



James Hogge 
Peabody College of Vanderbilt University 

John Murre7-1 
Homerton College, The University of Cambridge 



tPuCATlONAL «t PURGES iNfOPMATlQN 
CFNTERiERO 

Minor changes hflH> twe" mdiiv lo <mtvi)^^> 



• Points of viff* Of osxni. ,ns«rftti»0infh,»t5o(y 
0£Ri pofijt*cn Of poNcy 



• PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 



Running Head: Exploring Values 



Presented at the annual meeting of the Mid-South Educational 
Research Association, Lexington, KY, November 15, 1991. The 
authors wish to express their thanks to Ms. Oraphin Choochom for 
her assistance with the data analyses. 



2 

BEST COPY AVAILABLE 



Explor..ng Values 



Abstract 



This research examined individual differences with respect to 
values underlying the assessment of teaching competence. This is 
an important area of investigation because explication of such 
values should enhance the effectiveness and fairness of assess- 
ment. 

Seven classroom teachers and five teacher educators indicated 
the relative importances (expressed values) they attached to six 
aspects of teaching and independently rated the overall competence 
of 50 hypothetical student teachers for whom scores on the six 
aspects i'«»re available. Relative weights (implemented values) 
applied to the six aspects were then computed for each participant 
from a regression model using scores on the six aspects as 
predictors and the participant's overall ratings as criterion. 

Individual differences with respect to both expressed and 
implemented values were found; furthermore, several participants 
exhibited discrepancies between their expressed values and their 
own implemented values. This latter result may reflect a lack of 
self awareness and could adversely affect communication among 
assessors and between assessor and assessee. 
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Exploring Values Underlying the Assessment of Teaching Competence: 
An Application of Social Judgment Theory 



The assessment of teaching competence requires the formation 
of glolsal judgments based upon several variables. For example, 
the handbook provided for preservice teacher education students at 
a major British university suggests that the evaluation of their 
performance as student teachers by university teacher education 
faculty and classroom teachers supervising their student teaching 
occurs within a framework including six aspects of teaching: (1) 
Teacher/Pupil Relationships, (2) Preparation and Management, (3) 
Curriculum Content, (4) Classroom Interaction, (5) Assessment and 
Records, and (6) Self -Evaluation. An explanation of tnese aspects 
appears in Table 1. The handbook offers them as an aid for 
organizing evaluative discussions between students and those 
responsible for guiding their practice teaching. 

According to Social Judgment Theory (Brehmer & Joyce, 1988b; 
Hammond, McClelland, & Mumpower, 1980) , information about these 
aspects may be viewed as cues in Brunswik's (Brunswik, 1955) lens 
model. As shown in Figiire 1, evaluative judgments are estimates 
of an unknowable distal variable and must necessarily be based 
upon information available in the cues (Hogge, Fellendorf , Moore, 
& Wuescher, 1979) . Differential weighting f utilization) of the 
cues by a particular judge is likely as his or her values come 
into play. 

An individual's judgment system (and, hence, values) may be 
modeled with multiple linear regression (Adelman, 1988; Hammond & 
Adelman, 1976) , yielding the following equation: 

In the above equation Y corresponds to an individual's judgments 
(estimates of the distal variable, professional competence) , is 
a particular cue (aspect of teaching) , and is the corresponding 
relative weight (Hoffman, 1960) . Within this model the set of 
also estimates the individual's implemented values with respect to 
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the judgment task. 

Of course, these same individuals attempt to communicate 
their values in a direct manner during the supervision process. 
Conversations within the framework of the six aspects of teaching 
could reasonably be expected to include statements about which 
aspects of professional coi^tence are considered most important, 
thus leading a student teacher to emphasize a subset of the 
aspects in his or her own efforts to receive a favorable 
evaluation. 

Unfortunately, studies involving various populations dealing 
with a variety of variables have shown that individuals called 
upon to make global judgments on the basis of several variables 
(cues) typically report patterns of information usage differing 
from those exhibited in their ratings of real or hypothetical 
cases (Brehmer & Brehmer, 1988a). In other words, their ^xprgg^ed 
values (statements about the relative importances of cues) tend to 
differ from their implemented values (the relative weights 
computed from a regression model of their judgments) . 

Because such discrepancies could handicap communication 
between student teachers and their mentors, introduce an element 
of unfairness to the assessment process, and thus diminish the 
effectiveness of their preservice educatior , it seemed important 
to explore (1) whether there are, for parti ::ular supervisors of 
practice teaching, discrepancies between expressed and implemented 
values and (2) whether there are individual differences with 
respect to expressed and implemented values. 

Method 

P^yti<?ip4^ts 

The participants in this study were nine classroom teachers 
preparing to serve as school-based mentors of preservice teachers 
during their practice teaching and five teacher educators on the 
faculty of the major British university attended by the preservice 
teachers . 

Procedure 

Each participant was given definitions of the six aspects of 
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teaching competence and asked to apportion 100 points among the 
six aspects to indicate their relative importances (see Cook & 
Stewart, 1975) . These relative weights were assumed to estimate 
the expressed values of the participants. 

Next, each participant was asked to provide an overall rating 
of the professional compete^'ce of each of 50 hypothetical student 
teachers. The performance of each student teacher with respect to 
the six aspects of teaching competence was summarized on a 
separate sheet of paper in a randomly-generated (by the computer 
program POLICY PC, Executive Decision Services, 1988) profile like 
the example in Figure 2. The same rating scale (shown in Figure 
2) was used for both the summary of the performance of each 
student teacher and for the overall rating supplied by each 
participant - 

Results 

As can be seen in Table 2, correlations among the cues 
(aspects of teaching competence) were generally quite low, as 
would be expected for randomly-generated variables. Correlations 
among the ratings by the classroom teachers (Table 3, median r = 
0.73) and among the ratings by the university teacher educators 
(Table 4, median r = 0.65) were generally moderate to high and in 
the same range as the correlations between ratings by the same two 
groups (Table 5, median r = 0.70). In general, the between-groups 
rater agreement seems about as high as the agreement within either 
group. 

The two groups were also approximately equivalent with 
respect to how well their ratings could be modeled with strictly 
linear multiple regression equations. Table 6 contains the 
squared multiple correlations (R-squares) of those equations. In 
the regression equation for each participant his or her ratings 
served as the criterion variable and scores on the six aspects of 
teaching competence were the predictors. The mean R-squares of 
the classroom teachers and the university faculty members were 
0.79 and 0.74, respectively, and were not significantly different 
at the 0.05 level. 

The relative weights (implemented values) computed from each 
participant's regression equation and tne allocations of 100 
points among the six aspects (expressed values) were less similar 
for the two groups. Although both groups tended to indicate, on 
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average, that they deemed the six aspects to be of roughly equal 
importance, the relative weights from the participants' regression 
equations produced aggregate importance profiles (Tables 7 and 8) 
that were not so nearly flat. Among the classroom teachers, 
Preparation and Management (mean implemented value, 20.8) and 
Classroom Interaction (21.7) were most emphasized, while 
Self -Evaluation (9.3) emerged as least important in actual 
ratings. In making their ratings, the university teacher 
educators tended to draw most heavily upon Curriculum Content 
(mean implemented value, 26.4) and Teacher/Pupil Relationship 
(22.6) while clearly de-emphasizing Assessment and Records (6. 0} . 
In both groups individual differences with respect to expressed 
values (reflected in standard deviations, minima, and maxima) 
tended to be less dramatic than individual differences with 
respect to implemented values. 

Mean absolute differences between expressed and implemented 
values (Table 9) tended to be higher for the university teacher 
educators than for the classroom teachers (mean total 
di;9crepancies =28.4 and 44.0, respectively; separate variances t 
= 2.26, p < 0.08), although severe inequality of sample variances 
(221.0 and 57.6, respectively; F[4.8] = 3.837, p * 0.05) and the 
very small size of the sample of university teacher educators 
cloud the picture. Never ' heless, it is clear that the 
participants in this study exhibited a rather wide range (16 to 
60) of discrepancies between expressed and implemented values. 

Discussion 

The present study provides two alternative descriptions of 
the assessment of professional competence that paint portraits of 
the same subject in differing detail. The first, and most tra- 
ditional, is based upon the examination of rater agreement and 
suggests that although the general level of consensus is modest, 
the ratings of the classroom teachers and the university teacher 
educators who participated in this study are roughly equivalent. 
Unfortunately, if considered alone, rater agreement says nothing 
about the value systems underlying the ratings. 

The second description focuses upon each rater's use of 
available information and offers a profile of his or her values 
(relative weights applied to aspects of professional competence) 
implemented in making judgments on a case-by-case basis. The same 
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description also includes a profile of values expressed prior to 
making case-by-case judgments. These expressed values would 
presumably be reflected in statements a rater might during 
supervision of a student's practice teaching and would be taken as 
indications of what the rater could be expected to emphasize in 
subseg[uent assessments of the student's performance. 
Discrepancies between the rater's expressed and implemented values 
would thus tend to disrupt communication between the rater and the 
student and contribute to dissatisfaction with the assessment 
process . 

Although the results of this study suggest comparatively 
minor individual differences with respect to expressed values (the 
six aspects of teaching competence were typically characterized as 
being of roughly equal importance) , there were comparatively large 
individual differences with respect to implemented values. In 
other words, a student teacher would probably hear rather similar 
statements about the relative importance of the six aspects from 
almost any of the participants in this study, but tv»e same student 
could encounter widely differing patterns of implc ^lented values at 
the time of performance assessment. 

The range of total discrepancies between expressed and 
implemented values suggests that the student could also experience 
differing degrees of consistency, from rater to rater, between 
statements about the relative importance of the aspects and 
aspects actually emphasized in assessment of the student's perfor- 
mance. Increased self -insight on the pa^t of the rater and more 
accurate communication os: values could improve both supervision 
and assessment. 

Future applications of the Social Judgment Theory paradigm to 
this line of research should investigate in more detail the conse- 
quences of various levels of discrepancy between expressed and 
implemented values and whether detailed cognitive feedback 
(Balzer, Doherty, & O'Connor, 1989; Cooksey & Freebody, 1986) can 
improve awareness of one's implemented values and reduce discrep- 
ancies between expressed and implemented values. Some (e.g., 
Fischhoff , 1991; Fischhoff , Slovic, & Lichtenstein, 1980) would 
suggest that it is primarily expressed values that are labile and 
therefore most likely to change, while others (e.g., Brehmer & 
Brehmer, 1988; Hammond & Grassia, 1985) would also expect changes 
in implemented values. But regardless of how they come about. 
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effective communication about values and consistency of word and 
deed in the assessment of professional competence are worthwhile 
goals to pursue. 
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Figt're 1 : Evaluation of Teachmg in Terms of Brunswik*s Lens Model 
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Figure 2: Profile of Performance of Hypothetical Student Teacher 

STUDENT TEACHER NUMBER 1 

Teacher/Pupil Relationships 
Preparation and Management 
Curriculum Content 
Classroom Interaction 
Assessment and Record^ 
Self-Evaluation 



5 Distinction 

5 Distinction 

2 Borderline 

3 Pass 

4 Credit 
3 Pass 



Overall Competence 
(Please circle 
your rating) 



5 Distinction 

4 Credit 

3 Pass 

2 Borderline 

1 Fail 
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Table 1 

Some Aspects of Teaching Competence 



Aspegt 



Features 



Teacher / Pup i 1 
Relationships 



Manner toward pupils and their response. 
Evidence of )cnowing pupils as individuils and 
the extent to which their concerns are understood. 
Fostering appropriate behavior and self-discipline 
in children. 



Preparation and 
Management 



Ability to: 

Specify intentions compatible with the school's 
curriculum guidelines and work flexibly toward 
them. 

Plan and justify schemes of work and individual 
sessions appropriate to the abilities, experiences 
and concerns of pupils. 

Match the activities to the children's learning. 
Develop links across the curriculum, to show 
continuity and progression in the children's 
learning experiences. 

Manage the grouping and movement of children 
appropriate to the activity. 
Maximize use of time and space. 

Organize work in specialist areas where particular 
attention should be paid to safety factors. 
Manage resource materials and equipment. 
Produce quality materials, e.g., worksheets, cards, 
transparencies . 

Display and present children's vc-^ -effectively. 
Manage the movement of children around and outside 
the school. 



(Continued on next page) 
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Table 1, Continued 



Aspect 



Features 



Curriculum 
Content 



Knowledge and understanding of the appropriate 
curriculum content. 

Ability to select concepts, skills and procedures 
at an appropriate level for the pupils. 
Appreciation of subject areas as part of the whole 
school curriculum. 

Ability to justify the schemes of work and the 
choice of content for each session. 



Classroom 
Interaction 



Assessment 
and Records 



Ability to: 

Gain and hold attention, stimulating the motivation 
to learn. 

Present ideas and information, together with appro- 
priate explanations, demonstrations and feedback. 
Manage discussion, the exchange and use of ideas 
and responses of pupils. 
Use appropriate questioning techniques. 
Use voice and gesture effectively. 
Adopt a variety of teaching strategies. 
Monitor pupils' responses and adjust teaching 
accordingly. 

Assessment of pupils' progress within a session and 
over a longer period of time. 
Attempts at individual diagnosis of the diffi- 
culties met by individual pupils. 

Ability to record pupils' achievements and progress 
and to report effectively on these. 



Self -Evaluation 



Ability to: 

'Monitor and develop a reflective approach to one's 
own teaching performance and learn from experience, 
Evaluate sessions in all curriculum areas and 
schemes of work. 

Respond professionally to advice. 
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Table 2 

Correlati ons Among Cues 







1 


2 


3 


4 


5 


6 


1. 


Teacher/Pupil 
Relationships 


1.00 


.14 


-.05 


.20 


.29 


.12 


2. 


Preparation and 
Management 


.14 


1.00 


-.02 


-.04 


.20 


-.18 


3. 


Curriculum 
Content 


-.05 


-.02 


1.00 


-.05 


-.15 


-.02 


4. 


Classroom 
Interaction 


.20 


-.04 


-.05 


1.00 


.11 


-.00 


5. 


Assessment and 
Records 


.29 


.20 


-.15 


.11 


1.00 


-.03 


6. 


Self -Evaluation 


.12 


-.18 


-.02 


-.00 


-.03 


1.00 



Note: Values for each cue were randomly-generated integers 

ranging from 1 to 5; 50 profiles of hypothetical student 
teachers were thus generated. 
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Table 3 

Correlations Among Ratings bv Classroom Teachers 





1 


2 


3 


4 


5 


6 


7 


8 


9 


1 


1.00 


.77 


.72 


.81 


.74 


.78 


.73 


.62 


.72 


2 


.77 


1.00 


.73 


.86 


.76 


.81 


.80 


.73 


.86 


3 


.72 


.73 


1.00 


.77 


.71 


.71 


.65 


.54 


.65 


4 


.81 


.86 


.77 


1.00 


.73 


.81 


.73 


.72 


.84 


5 


.74 


.76 


.71 


.73 


1.00 


.77 


.68 


.67 


.70 


6 


.78 


.81 


.71 


.81 


.77 


1.00 


.76 


.73 


.73 


7 


.73 


.80 


.65 


.73 


.68 


.76 


1.00 


.78 


.75 


8 


.62 


.73 


.54 


.72 


.67 


.73 


.73 


1.00 


.72 


9 


.72 


.86 


.65 


.84 


.70 


.73 


.75 


.72 


1.00 



Note: n = 50 profiles of hypothetical student teachers; 

for a = 0.05 (two-tailed), critical value of r = |0.276|. 
Median r = 0.73. 
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Table 4 

Correlations Among Ratings by University Teacher Eaucators 





1 


2 


3 


4 


5 


1 


1.00 


.64 


.82 


.57 


.70 


2 


.64 


1.00 


.59 


.72 


.77 


3 


.82 


.59 


1.00 


.57 


.64 


4 


.57 


.72 


-.57 


1.00 


.66 


5 


.70 


.77 


.64 


.66 


1.00 



Note: n = 50 profiles of hypothetical student teachers; 
for a = 0.05 (two-tailed), critical r « |0.276|. 
Median r = 0.65. 
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Table 5 

Correlations Betwee n Ratings bv ClassrooiB Teachers and 
Ratings bv University Teacher Educators 



University Teacher Educators 



Classroom Teacher 1 



75 .70 .81 .60 



.66 



Classroom Teacher 2 



.91 .69 



.76 



.56 .73 



Classroom Teacher 3 



.70 



.65 



.68 



.66 



.66 



Classroom Teacher 4 



cBl 



.72 



.77 .63 



76 



Classroom Teacher 5 



70 



.70 



.68 



.63 



.63 



Classroom Teacher 6 



80 



.72 



.70 



.62 



.77 



Classroom Teacher 7 



78 



.61 



.74 



.57 



.68 



Classroom Teacher 8 



69 



.70 



.72 



.51 



.65 



Classroom Teacher 9 



79 



.62 



72 



37 



.60 



Note: n = 50 profiles of hypothetical student teachers; 
for a = 0.05 (two-tailed), critical r = |0.276|. 
Median r = 0.70. 
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Table 6 

Squared Multiple Co rrelations of Regression Equations 
Used to Mo del Ratings bv Participants 



Classroom Teacher 1^-Squa^^ 



1 


0.7293 


2 


0.8566 


3 


0.7598 


4 


0.8308 


5 


0.7412 


6 


0.8734 


7 


0.7464 


8 


0.7001 


9 


0.8503 



Mean = 0.7875 



Teacher Educator R-Souare 

1 0.8091 

2 0.7636 

3 0.7816 

4 0.7144 

5 0.6553 Mean = 0.7448 



Note: For each participant, criterion = ratings of 50 
hypothetical student teachers and predictors = 
six cues. All R-squares are significant at the 
a = 0.05 level; the difference between the means 
is nonsignificant. 
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Table 7 

Ex pressed and Implem entied Values of Classroom Teach.egg 





Expressed Values 




Implemented Values 




Mean 


S.D. 


Min 


Max 


Mean 


S.D. 


Min Max 


Teacher / Pup i 1 


17 . 3 


2.7 


13 


20 


14.5 


4.1 


10 20 


Relationships 
















Preparation and 


20.2 


2.1 


17 


25 


20.8 


5.7 


14 32 


Management 
















Curriculum 


15.8 


3.0 


10 


20 


16.3 


6.8 


2 23 


Content 
















Classroom 


17.9 


3.7 


13 


25 


21.7 


6.8 


8 32 


Interaction 
















Assessment and 


14.3 


2.0 


10 


17 


12.9 


4.3 


6 18 


Records 
















Self -Evaluation 


14.5 


4.1 


10 


20 


9.3 


3.6 


2 15 



Note: Expressed values = allocation of 100 points among aspects; 
Implemented values = relative weights (scaled to total 100 
points) from regression model of ratings of hypothetical 
student teachers' professional competence. 
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Table 8 

Evpressed and Imple m ented Values of University Teacher Educatoyp 



Expressed Values Implemented Values 



Mean S.D. Min Max Mean S.D. Min Max 

Teacher/Pupil 23.0 4.5 20 30 22.6 9.4 12 33 

Relationships 



Preparation and 20.8 2.3 19 25 15.0 8.2 
Management 



2 22 



Curriculum 17.8 4.8 13 25 26.4 14.0 12 45 
Content 

Classroom 19.6 3.6 15 25 19.8 2.6 16 23 
Interaction 

Assessment and 13.0 7.4 5 25 6.0 3.2 2 10 
Records 

Self -Evaluation 15.8 7.5 5 25 10.6 5.2 3 17 



Note: Expressed values = allocation of 100 points among aspects; 
Implemented values = relative weights (scaled to total 100 
points) from regression model of ratings of hypothetical 
student teachers' professional competence. 
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Table 9 

Discrepancies Between Expressed and Implemented Values 





Classroom 


Teachers 


Teacher Educators 




Mean 


S.D. 


Min 


Max 


Mean 


S.D. 


Hin Max 


Teacher/ Pupil 
Relationships 


3.7 


3.1 


0 


9 


9.6 


2.7 


6 13 


Preparation and 
Management 


3.5 


3.6 


0 


12 


6.2 


7.2 


1 18 


Curriculum 
Content 


5.4 


3.9 


0 


13 


9.8 


10.7 


1 25 


Classroom 
Interaction 


6.7 


5.3 


1 


17 


4.2 


2.4 


0 6 


Assessment and 
Records 


4.0 


3.0 


1 


9 


9.0 


6.5 


2 19 


Self -Evaluation 


5.1 


2.9 


0 


9 


5.2 


5.3 


1 14 


Total Discrepancy 


28.4 


7.6 


16 


38 


44.0 


14.9 


27 60 



Note: Discrepancy « absolute value of difference between 
corresponding expressed and implemented values. 
Groups « nine classroom teachers and five university 
teacher educators. 
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